Hello,

we've currently got a rather odd problem with our Exchange 2010 Server (Sp3 RU8):

During business hours, the RPC Client Access Process (Microsoft.Exchange.RpcClientAccess.Service) hits 100% CPU load on the Exchange Server. No matter what we have tried, the problem has so far refused to go away.

Additional symptoms:

• The load is caused by few (3-5) Mailboxes at a given time
• The Mailboxes causing this vary each day, can hit any Mailbox with no pattern discernible

What we've tried so far without success:

• Stop/Rebuild Searchindex
• Move all Mailboxes, dropping bad items
• Move (sole) public folder database
• Migrate Public folder database
• Stop other client access methods (Active Sync)
• Ensure throttling is in place
• Ensure Public Folders are configured properly and that the current database is accepted as the default database
• Throw excessive hardware at it
• Build a new Exchange Server and move content over
• Rebuild Virtual Directories
• Maximized logging of all kind (logs say everything is great)

In the RPC Access logs, virtually all errors are generic as expected during daily operations:

AccessDenied, Collision, Object Changed, ...

The only clue I have left - but cannot find any information on - is this:

There is one kind of Operation that takes an incredible amount of time:
Disconnect
We have clients (All of them using Outlook 2010) that spend literally days disconnecting.

But what does that tell me? How can I dig deeper into this?
If that's not it: What else can I try?

Cheers and thanks in advance for any insights,

Hello

tip: https://support.microsoft.com/en-us/kb/2497453
and check client usage with Log Parser Studio  https://gallery.technet.microsoft.com/office/Log-Parser-Studio-cd458765

Hi,

Please install a new CAS server and deploy CAS array to load the service request. In Event Viewer, please collct more error logs for further troubleshooting.

Additionally, the Exchange 2010 SP3 RU9 is released. Maybe we can upgrade the Exchange server to the latest version to have a try.

Regards,

Hi Winnie,

thanks for the headsup on CU9. We'll see to its installation as soon as we manage to get the downtime (not today probably). It doesn't list this issue on the "fixed" list, but I don't see how it could hurt giving it a shot.

We had the CAS array-ed during migration to the second Exchange (with both Exchange Servers being CPU-capped).

In the mean time:
Just noticed I'd never taken a look at the the process with Process Monitor (shame on me). I did and the results were ... interesting.

The Exchange Server appears to be in an infinite loopback loop (~385000 Connection to itself within 7.5 Minutes, all on the same receiving port. Various but very few sending ports. Guessing here, but I'd assume one Port per problematic connected Mailbox). The Ports are all in the high ports (Receiving Port is currently 33430, but that'll probably change during reboots).

We further had some indications that this probably is a Public Folder related issue. We can exclude bad items as the cause, and while the general structure is an ancient one with many migrations, I'm fairly confident we got any ghosts killed. Any ideas what else I might be able to try?

Cheers,
Fred